Department of Econometrics and Business Statistics
Recap
What is relational data?
How to combine different data sets for data analysis?
Joins and Keys
Different types of joins
Stacking data frames
Working with Excel files
Outline
What is web scraping?
rvest and polite
What is a function?
File paths and RStudio projects
Web Scraping
Web Scraping
An increasing amount of data is available on the web, but they are often provided in an unstructured format.
Copying and pasting is always an option, but it’s time-consuming and prone to errors.
Web Scraping
Web scraping is the process of extracting data from websites automatically and transform it into a structured dataset.
There are two major types of web scraping:
Screen Scraping
Extracting data from the HTML content of a web page, with HTML parser or regular expression matching.
API (application programming interface) Data Retrieval
Website offers a set of structured http requests that return JSON or XML files.
R provides all the essential tools needed for web scraping, along with the advantage of direct data analysis. However, other languages like Python, Perl, and Java are also effective options.
Hypertext Markup Language (HTML)
HTML is the standard markup language for documents designed to be displayed in a web browser.
It defines the content and layout of web pages in a hierarchical, tree-like structure.
Here is a basic example of an HTML document:
<html><head><title>This is a title</title></head><body><p style="font-size: 30px;"> ETC1010</p></body></html>
It does not work well on websites with dynamically generated content (e.g. IMDB)!
Usage
A box will open in the bottom right of the website. Click on a page element that you would like your selector to match (it will turn green). SelectorGadget will then generate a minimal CSS selector for that element, and will highlight (yellow) everything that is matched by the selector.
Now click on a highlighted element to remove it from the selector (red), or click on an unhighlighted element to add it to the selector. Through this process of selection and rejection, SelectorGadget helps you come up with the appropriate CSS selector for your needs.
Google Chrome DevTools
You can right-click on any element on a webpage and choose “Inspect” to open DevTools.
In DevTools, click on the top-left icon that looks like a square with a cursor inside.
Hover over any part of the webpage, a tooltip will appear, showing the CSS selector.
Top 250 Movies on IMDB
Let’s get titles of top 250 movies.
Each title is in an <h3> element with the class ipc-title__text.
[1] "IMDb Charts"
[2] "1. The Shawshank Redemption"
[3] "2. The Godfather"
[4] "3. The Dark Knight"
[5] "4. The Godfather: Part II"
[6] "5. 12 Angry Men"
[7] "6. Schindler's List"
[8] "7. The Lord of the Rings: The Return of the King"
[9] "8. Pulp Fiction"
[10] "9. The Lord of the Rings: The Fellowship of the Ring"
[11] "10. The Good, the Bad and the Ugly"
[12] "11. Forrest Gump"
[13] "12. The Lord of the Rings: The Two Towers"
[14] "13. Fight Club"
[15] "14. Inception"
[16] "15. Star Wars: Episode V - The Empire Strikes Back"
[17] "16. The Matrix"
[18] "17. GoodFellas"
[19] "18. One Flew Over the Cuckoo's Nest"
[20] "19. Interstellar"
[21] "20. Se7en"
[22] "21. It's a Wonderful Life"
[23] "22. Seven Samurai"
[24] "23. The Silence of the Lambs"
[25] "24. Saving Private Ryan"
[26] "25. City of God"
[27] "Recently viewed"
Note
There are only 25 movies because the content on the webpage is dynamically generated by JavaScript.
read_html_live()
read_html_live() runs a live web browser (Chrome) in the background.
It allows you to access dynamically generated HTML elements and interact with the live page, such as clicking buttons or typing in forms.
[1] "IMDb Charts"
[2] "1. The Shawshank Redemption"
[3] "2. The Godfather"
[4] "3. The Dark Knight"
[5] "4. The Godfather Part II"
[6] "5. 12 Angry Men"
[7] "6. Schindler's List"
[8] "7. The Lord of the Rings: The Return of the King"
[9] "8. Pulp Fiction"
[10] "9. The Lord of the Rings: The Fellowship of the Ring"
[11] "10. The Good, the Bad and the Ugly"
[12] "11. Forrest Gump"
[13] "12. The Lord of the Rings: The Two Towers"
[14] "13. Fight Club"
[15] "14. Inception"
[16] "15. Star Wars: Episode V - The Empire Strikes Back"
[17] "16. The Matrix"
[18] "17. GoodFellas"
[19] "18. One Flew Over the Cuckoo's Nest"
[20] "19. Interstellar"
[21] "20. Seven"
[22] "21. It's a Wonderful Life"
[23] "22. Seven Samurai"
[24] "23. The Silence of the Lambs"
[25] "24. Saving Private Ryan"
[26] "25. City of God"
[27] "26. Life Is Beautiful"
[28] "27. The Green Mile"
[29] "28. Terminator 2: Judgment Day"
[30] "29. Star Wars: Episode IV - A New Hope"
[31] "30. Back to the Future"
[32] "31. Spirited Away"
[33] "32. The Pianist"
[34] "33. Parasite"
[35] "34. Psycho"
[36] "35. Gladiator"
[37] "36. The Lion King"
[38] "37. Spider-Man: Across the Spider-Verse"
[39] "38. The Departed"
[40] "39. Whiplash"
[41] "40. American History X"
[42] "41. Leon"
[43] "42. Grave of the Fireflies"
[44] "43. The Prestige"
[45] "44. Harakiri"
[46] "45. Dune: Part Two"
[47] "46. The Usual Suspects"
[48] "47. Casablanca"
[49] "48. Untouchable"
[50] "49. Cinema Paradiso"
[51] "50. Modern Times"
[52] "51. Alien"
[53] "52. Rear Window"
[54] "53. Once Upon a Time in the West"
[55] "54. City Lights"
[56] "55. Django Unchained"
[57] "56. Apocalypse Now"
[58] "57. Memento"
[59] "58. WALL·E"
[60] "59. Raiders of the Lost Ark"
[61] "60. 12th Fail"
[62] "61. The Lives of Others"
[63] "62. Sunset Blvd."
[64] "63. Avengers: Infinity War"
[65] "64. Paths of Glory"
[66] "65. Spider-Man: Into the Spider-Verse"
[67] "66. Witness for the Prosecution"
[68] "67. The Shining"
[69] "68. The Great Dictator"
[70] "69. Aliens"
[71] "70. Inglourious Basterds"
[72] "71. The Dark Knight Rises"
[73] "72. Coco"
[74] "73. Amadeus"
[75] "74. Dr. Strangelove or: How I Learned to Stop Worrying and Love the Bomb"
[76] "75. Toy Story"
[77] "76. Oldboy"
[78] "77. American Beauty"
[79] "78. Avengers: Endgame"
[80] "79. Braveheart"
[81] "80. Das Boot"
[82] "81. Good Will Hunting"
[83] "82. Princess Mononoke"
[84] "83. Joker"
[85] "84. Your Name."
[86] "85. High and Low"
[87] "86. 3 Idiots"
[88] "87. Once Upon a Time in America"
[89] "88. Singin' in the Rain"
[90] "89. Capernaum"
[91] "90. Come and See"
[92] "91. Requiem for a Dream"
[93] "92. Toy Story 3"
[94] "93. Star Wars: Episode VI - Return of the Jedi"
[95] "94. Eternal Sunshine of the Spotless Mind"
[96] "95. The Hunt"
[97] "96. 2001: A Space Odyssey"
[98] "97. Ikiru"
[99] "98. Reservoir Dogs"
[100] "99. Lawrence of Arabia"
[101] "100. The Apartment"
[102] "101. Incendies"
[103] "102. Oppenheimer"
[104] "103. North by Northwest"
[105] "104. Scarface"
[106] "105. Citizen Kane"
[107] "106. Double Indemnity"
[108] "107. M"
[109] "108. Vertigo"
[110] "109. Full Metal Jacket"
[111] "110. Heat"
[112] "111. Amélie"
[113] "112. Up"
[114] "113. A Clockwork Orange"
[115] "114. To Kill a Mockingbird"
[116] "115. A Separation"
[117] "116. The Sting"
[118] "117. Die Hard"
[119] "118. Indiana Jones and the Last Crusade"
[120] "119. Like Stars on Earth"
[121] "120. Metropolis"
[122] "121. Snatch"
[123] "122. 1917"
[124] "123. L.A. Confidential"
[125] "124. Bicycle Thieves"
[126] "125. Hamilton"
[127] "126. Taxi Driver"
[128] "127. Downfall"
[129] "128. Dangal"
[130] "129. Batman Begins"
[131] "130. For a Few Dollars More"
[132] "131. The Wolf of Wall Street"
[133] "132. Green Book"
[134] "133. Some Like It Hot"
[135] "134. The Kid"
[136] "135. Judgment at Nuremberg"
[137] "136. The Truman Show"
[138] "137. The Father"
[139] "138. All About Eve"
[140] "139. Shutter Island"
[141] "140. There Will Be Blood"
[142] "141. Top Gun: Maverick"
[143] "142. Jurassic Park"
[144] "143. Casino"
[145] "144. Ran"
[146] "145. The Sixth Sense"
[147] "146. Pan's Labyrinth"
[148] "147. Unforgiven"
[149] "148. No Country for Old Men"
[150] "149. The Thing"
[151] "150. A Beautiful Mind"
[152] "151. Kill Bill: Vol. 1"
[153] "152. The Treasure of the Sierra Madre"
[154] "153. Yojimbo"
[155] "154. The Great Escape"
[156] "155. Monty Python and the Holy Grail"
[157] "156. Finding Nemo"
[158] "157. Prisoners"
[159] "158. Howl's Moving Castle"
[160] "159. Rashomon"
[161] "160. The Elephant Man"
[162] "161. Dial M for Murder"
[163] "162. Chinatown"
[164] "163. Gone with the Wind"
[165] "164. Lock, Stock and Two Smoking Barrels"
[166] "165. The Secret in Their Eyes"
[167] "166. Inside Out"
[168] "167. V for Vendetta"
[169] "168. Raging Bull"
[170] "169. Three Billboards Outside Ebbing, Missouri"
[171] "170. Trainspotting"
[172] "171. The Bridge on the River Kwai"
[173] "172. Klaus"
[174] "173. Catch Me If You Can"
[175] "174. Fargo"
[176] "175. Warrior"
[177] "176. Spider-Man: No Way Home"
[178] "177. Gran Torino"
[179] "178. Harry Potter and the Deathly Hallows: Part 2"
[180] "179. Million Dollar Baby"
[181] "180. My Neighbour Totoro"
[182] "181. Mad Max: Fury Road"
[183] "182. Children of Heaven"
[184] "183. Ben-Hur"
[185] "184. 12 Years a Slave"
[186] "185. Before Sunrise"
[187] "186. Blade Runner"
[188] "187. Barry Lyndon"
[189] "188. The Grand Budapest Hotel"
[190] "189. Hacksaw Ridge"
[191] "190. Gone Girl"
[192] "191. Dead Poets Society"
[193] "192. Maharaja"
[194] "193. Memories of Murder"
[195] "194. In the Name of the Father"
[196] "195. The Gold Rush"
[197] "196. Monsters, Inc."
[198] "197. Wild Tales"
[199] "198. The Deer Hunter"
[200] "199. The General"
[201] "200. Jaws"
[202] "201. Sherlock Jr."
[203] "202. How to Train Your Dragon"
[204] "203. Ratatouille"
[205] "204. On the Waterfront"
[206] "205. Mary and Max"
[207] "206. The Wages of Fear"
[208] "207. The Third Man"
[209] "208. Wild Strawberries"
[210] "209. Le Mans '66"
[211] "210. Mr. Smith Goes to Washington"
[212] "211. Tokyo Story"
[213] "212. Logan"
[214] "213. Rocky"
[215] "214. The Big Lebowski"
[216] "215. The Seventh Seal"
[217] "216. Room"
[218] "217. Spotlight"
[219] "218. The Terminator"
[220] "219. Hotel Rwanda"
[221] "220. Platoon"
[222] "221. La haine"
[223] "222. Pirates of the Caribbean: The Curse of the Black Pearl"
[224] "223. Before Sunset"
[225] "224. The Passion of Joan of Arc"
[226] "225. Jai Bhim"
[227] "226. The Best Years of Our Lives"
[228] "227. The Exorcist"
[229] "228. Rush"
[230] "229. The Incredibles"
[231] "230. Network"
[232] "231. The Wizard of Oz"
[233] "232. Stand by Me"
[234] "233. Hachi: A Dog's Tale"
[235] "234. The Sound of Music"
[236] "235. My Father and My Son"
[237] "236. The Handmaiden"
[238] "237. To Be or Not to Be"
[239] "238. Into the Wild"
[240] "239. The Battle of Algiers"
[241] "240. Groundhog Day"
[242] "241. The Grapes of Wrath"
[243] "242. The Iron Giant"
[244] "243. Amores perros"
[245] "244. Rebecca"
[246] "245. The Help"
[247] "246. Cool Hand Luke"
[248] "247. It Happened One Night"
[249] "248. Paris, Texas"
[250] "249. Aladdin"
[251] "250. Scent of a Woman"
[252] "You have rated"
[253] "More to explore"
[254] "Charts"
[255] "Top Box Office (US)"
[256] "Most Popular Movies"
[257] "Top Rated English Movies"
[258] "Most Popular TV Shows"
[259] "Top 250 TV Shows"
[260] "Lowest Rated Movies"
[261] "Most Popular Celebs"
[262] "Movie News"
[263] "Top Rated Movies by Genre"
[264] "Recently viewed"
Note
Since we’re using a headless browser to access the website, it will execute the JavaScript and load all the dynamic content, similar to how you would see it when visiting the site with a standard browser.
Parsing the Raw Data
Find all real titles by matching strings that start with (^) a digit (\d) using a regular expression.
Match the pattern of starting digit(s), followed by a dot and a space, and replace them with an empty string.
Regular expressions are a concise and flexible tool for describing patterns in strings. See here for more.
# A tibble: 250 × 3
title rating year
<chr> <chr> <chr>
1 The Shawshank Redemption 9.3 1994
2 The Godfather 9.2 1972
3 The Dark Knight 9.0 2008
4 The Godfather Part II 9.0 1974
5 12 Angry Men 9.0 1957
6 Schindler's List 9.0 1993
7 The Lord of the Rings: The Return of the King 9.0 2003
8 Pulp Fiction 8.9 1994
9 The Lord of the Rings: The Fellowship of the Ring 8.9 2001
10 The Good, the Bad and the Ugly 8.8 1966
# ℹ 240 more rows
polite
Responsible Web Scraping
The polite package helps ensure responsible web scraping by following polite practices:
introducing yourself
asking for permission
scraping slowly
not making repeated requests
Responsible Web Scraping
Use bow() to introduce yourself to the website and check the robots.txt file to get permission to scrape.
<polite session> http://www.imdb.com
User-agent: polite R package
robots.txt: 35 rules are defined for 3 bots
Crawl delay: 5 sec
The path is scrapable for this user-agent
Programatically direct R to each page on the series_urls vector and run get_episode_number().
get_episode_number(series_urls[1])
# A tibble: 1 × 2
name episode_number
<chr> <int>
1 The Lord of the Rings: The Rings of Power 17
get_episode_number(series_urls[2])
# A tibble: 1 × 2
name episode_number
<chr> <int>
1 Kaos 8
get_episode_number(series_urls[3])
# A tibble: 1 × 2
name episode_number
<chr> <int>
1 Game of Thrones 74
Go to Each Page, Scrape Information
In other words, we want to map the get_episode_number() function to each element of series_urls.
This will hit the URLs one after another, and grab the information.
map_df() will combine all the tibbles into one.
map_df(series_urls, get_episode_number) %>%head()
# A tibble: 5 × 2
name episode_number
<chr> <int>
1 The Lord of the Rings: The Rings of Power 17
2 Kaos 8
3 Game of Thrones 74
4 Bad Monkey 10
5 Only Murders in the Building 50
File Paths and Project Organization
Please read this section on your own time
File Paths and Project Organization
It’s important when you start working on your own machine that you understand file storage hygiene.
It helps prevent unexpected problems and makes you more productive
You’ll spend less time fighting against strange file paths.
Not sure what a file path is? We will explain that as well!
What is A File Path?
This all might be a bit confusing if you don’t know what a file path is.
A file path is: “the machine-readable directions to where files on your computer live.”
“Set my working directory to this specific working directory”.
It means that you can read in data and other things like this:
data <-read_csv("data/mydata.csv")
Instead of
data <-read_csv("c:/really/long/file/path/to/this/directory/data/mydata.csv")
Using setwd()
This has the effect of making the file paths work in your file
This is a problem because, among other things, using setwd():
Has 0% chance of working on someone else’s machine (this includes you in >6 months)
Your file is not self-contained and portable. (Think: “What if this folder moved to /Downloads, or onto another machine?”)
To get this to work, you need to hand edit the file path to your machine.
This is painful. And when you do this all the time, it gets old, fast.
setwd() is banned by CRAN in package development!
What is the Alternative to setwd()?
I highly recommend when you start on a new idea, new research project, paper. Anything that is new. It should start its life as an RStudio project.
An RStudio project helps keep related work together in the same place. Amongst other things, they:
Keep all your files together
Set the working directory to the project directory
Starts a new session of R
Restore previously edited files into the editor tabs
Restore other rstudio settings
Allow for multiple R projects open at the same time.
RStudio Project
This helps keep you sane, because:
Your projects are each independent.
You can work on different projects at the same time.
Objects and functions you create and run from project idea won’t impact one another.
You can refer to your data and other projects in a consistent way.
And finally, the big one: RStudio projects help resolve file path problems, because they automatically set the working directory to the location of the RStudio project.
here
In some cases you might have many folders in your RStudio project.
To help navigate them appropriately, you can use the here package to provide the full path directory, in a compact way.
You can read the above here code as: In the folder data of the current RStudio project, there is a file called gapminder.csv, can you please give me the absolute path to that file?
here
This is really handy for a few reasons:
It makes things completely portable
Rmarkdown/Quarto documents have a special way of looking for files, this helps eliminate file path pain.
If you decide to not use RStudio projects, you have code that will work on any machine.